Some investigations into energy - based models

Some investigations into energy-based models Tijmen Tieleman Master of Science Graduate Department of Computer Science University of Toronto 2007 Three questions about various energy-based probability models are asked and answered. The first is whether the Contrastive Divergence algorithm computes the gradient of any function at all the answer is no. The second is whether there is a tractable Monte Carlo approximation to the gradient for variational learning in a large class of models including Sigmoid Belief Networks the answer is yes. The third is how we might do early stopping for Restricted Boltzmann Machines, which have intractable objective functions the problem is studied thoroughly, some old algorithms are reviewed and some new and better ones are introduced, and the way is pointed to the ultimate algorithm for this problem.